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ABSTRACT 



Kentucky's systemic reform initiative has been heralded as 
one of the more comprehensive and well -sustained reforms in recent history. 

To evaluate the course of this reform, the challenges associated with 
researching and evaluating a statewide systemic reform effort are detailed 
here. By using a description of the politics of evaluating systemic policy at 
the state level, the paper attempts to fill the gap between the investigation 
of how policy works and the investigation of how policy is evaluated. An 
embedded, single- case design was used to study the program over a 3 -month 
period. Examination of the system suggests that the political agenda and 
organizational structure of the Kentucky Department of Education influence 
evaluation processes. Such tension has been documented elsewhere, but the 
complexity of the reforms in the state prevents comprehensive answers to the 
question lawmakers want answered: Is this initiative effective? The findings 
indicate that the political boundaries of statewide policy-evaluation 
research are more complex and problematic than the generally intricate 
questions of reform effects. Moreover, it is argued that such political 
borders may prevent adequate assessment of reform efforts, thus forestalling 
improvements in teaching and learning. (Contain 37 references.) (RJM) 
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Purpose & Rationale 

Kentucky’s systemic reform initiative has been heralded as one of the most 
comprehensive and well-sustained reforms in recent history (ACCESS ERIC, 1994; Lindle, 

1995; Miller, 1994; Pipho, 1990; Quality Counts. 1997; Steffy, 1993). Due to its comprehensive 
nature, questions abound concerning the 1990 Kentucky Education Reform Act (KERA)’s effects 
on student learning (Bridge, Winograd & Petrosko, 1996; Guskey & Oldham, 1996; Lindle & 
Petrosko, in press). Converting those questions into researchable projects is not an easy task 
given the complexities of systemic reform. The purpose of this paper is to document the 
challenges associated with researching and evaluating a statewide systemic reform effort. 

The literature on the politics and process of evaluation and research on educational policy 
is quite extensive (Fischer & Forester, 1987; Guba and Lincoln, 1981, 1989; Hedrick, 1988; 
House, 1993; Palumbo, 1987; Patton, 1987; Weiss, 1970, 1987, 1988; Worthen & Sanders, 

1987). In contrast, literature documenting state-level reform is more focused on policy 
implementation processes rather than evaluation or research processes (Goertz, Floden, & 

O’Day, 1996; Kelley & Odden, 1995; King & Mathers, 1996; Marshall, Mitchell & Wirt, 1989; 
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Mitchell, Wirt, & Marshall, 1986). This paper attempts to fill the gap between the investigation 
of how policy works and the investigation of how policy is evaluated with a description of the 
politics of evaluating systemic policy at the state level. 

Data Sources & Methods 

The design of this study is what Yin (1987) termed the embedded, single case design. 

The embedded design is appropriate because the nature of systemic reform is a constant search 
for the fulcrum between state-level policy making/implementation and local policy 
making/implementation. Evaluation of systemic reform harbors the same kind of precarious 
equilibrium between subunits of analysis and the relationships and practices among them (Fiske, 
1994; Yin, 1987). Furthermore, the researchers were well-aware that Kentucky’s policy makers 
and implementers at all levels had a shifting focus from KERA’s components or strands to its 
synergy (Lindle & Petrosko, in press; Massed, Kirst & Hoppe, 1996). The single case design is 
justified by the unique nature of Kentucky’s context and policy efforts (Miles & Huberman, 

1994; Yin, 1987). 

Data were collected over several months in 1997. Sources included documentary searches 
of research and evaluation proposals and reports from state and private agencies and institutions 
of higher education. At this writing, not all the data had been collected and analyzed. 

Preliminary results for the extant data yielded a thematic description of the types of research 
designs applied to Kentucky’s reforms plus a list of potential interviewees (Holsti, 1969; 
Merriam, 1988). Three third-party evaluation projects were selected for this paper. One project 
had been established in 1991 and has operated continuously with the same evaluation 
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organization through the date of this research. A second project represents a shift from third 
party evaluation to internal evaluation back to external evaluation. The third project has been a 
continuous third party evaluation with several different parties receiving the contract. Themes 
concerning the politics of evaluation were developed from the documentary analysis (Guba & 
Lincoln, 1981; Hodder, 1994; Merriam, 1988; Miles & Huberman, 1994). 

Themes of the Politics of Kentucky’s Evaluation Process 

Several themes have emerged from the preliminary data analysis on these projects’ 
documents and administrative histories. Three of these themes are offered for discussion here. 
First, we describe how Kentucky’s evaluation needs have much in common with the literature on 
the relationships between practitioners and researchers in general. Next we discuss the ways in 
which the political context and pressures on the leadership in the Kentucky Department of 
Education influence the evaluation process. Finally, we describe how the comprehensive scope 
and design of Kentucky’s educational reform establishes its own constraints on evaluation. 

Kentucky exhibits typical tensions between practitioners and evaluators. The tensions 
between practitioners and evaluators are most apparent in the early discussions relative to the 
evaluation design itself. These discussions to renegotiate designs occurred in all three cases after 
the Request for Proposals (RFP)s’ responses have been evaluated and the contract let. The timing 
and the process create very different responses between evaluators and state agency personnel. 

In two of the cases, the timing of such discussions seemed surprising to some evaluators 
and researchers. In their eyes, their design won, but department officials asked the “winning” 
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researchers to modify their design based on what they saw in other proposals or other factors not 
specified in the RFP. Evaluators and researchers saw this as changing the economy of scale in 
the evaluation project. In two cases, evaluators did not allot “renegotiation” time or cost in their 
budget. In those cases, state agency personnel wanted to renegotiate design, but not time lines or 
budgetary costs. 

From the department’s perspective, the state agency program managers felt they need to 
gain more clarification of the research questions. Program managers have a vested interest in 
preserving the program as it currently exists and are resistant to the type of questions that might 
suggest there is another way of delivering or managing the program. Program managers have 
been charged, as most have been in Kentucky, with the statutory interpretation in the 
development of the specific components of the Kentucky Education Reform Act, including 
everything from describing the program to developing regulations and implementation strategies. 
Given their charge, program managers are more inclined to ask researchers to monitor 
implementation levels rather than evaluate program effects. Thus, RFPs suggest questions about 
effects, but the renegotiations often seem to researchers to imply a technical support role more 
appropriate to formative evaluation which department personnel might be expected to carry out 
as part of their job responsibilities. 

As the negotiations unfolded in all three cases, the project designs expanded. In one case 
the design expansion has been funded for seven successive years at $300,000 per year. In the 
other two cases, the expanded designs were not funded by the department. In one case, two 
universities provide the extra dollars. In the other case, the design collapsed back to the original 
proposal. Nevertheless, these repeated experiences prompted one university researcher to 
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eloquently mutter words about the Kentucky Department of Ed’s developing reputation for 
demanding “half-million dollar studies for $195.” 

While these tensions are not unique to Kentucky, they have deleterious effects on the 
studies in question. The expanded negotiations often inhibits the ability to deliver the expected 
product, at best. At worst, these tensions cloud the design itself. 

Political realities of KDE leadership drive evaluation design. State agency officials have 
been accused by partisan groups and stakeholders of being “cheerleaders” rather than objective 
evaluators of program effectiveness and efficiency. This criticism is not surprising, given the 
fact that KERA abolished the Kentucky Department of Education and reconfigured it during the 
first year after its enactment (Steffy, 1993; Van Meter, 1997). New staff were hired in key 
leadership positions. Many were from out-of-state. They were charged with development of 
various strands or components of KERA including preschool, primary, school-based decision 
making, extended school services, professional development, school improvement mechanisms 
for accountability, assessment and curriculum, and the continuing responsibilities for integrating 
federal programs such as special education and Title 1. With the overriding goal of successful 
implementation of the state’s massive systemic P-12 educational reform initiative, it is 
understandable that the developers and managers of the components are not viewed as objective. 

Whether appearing before a legislative committee, the Kentucky Board of Education, 
other state agencies, or evaluators from out of state, KDE officials found themselves in the 
position of promoting their program to establish its identity in the larger context of systemic 
reform. In all venues, KDE was asked to provide evidence supporting their claims. Early in the 
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reform, such requests could be put off with the very real condition that “It’s too early.” 

As time passed, pressure for evidence increased. Sometimes this led to sweeping 
generalizations based on anecdotes and limited observations without the benefit of a broad data 
base. The political ramifications of carving out such program-positive positions in a culture of 
demands for supporting data are akin to high-stakes/high-risk games. KDE officials have 
developed a love-hate relationship with data and people who either want their data or who 
produce data of their own. 

In two of the cases analyzed here, when researchers attempted to gather program 
effectiveness data, KDE offered a vast disclaimer that the data in question could not, and maybe 
should not, be disaggregated. In two cases, researchers were encouraged to collect data to answer 
the questions of interest. In one case, researchers were asked to provide secondary analysis of 
data, but data either were not as extensive as KDE believed or “not available.” 

There is no question that KERA has created a data-driven accountability culture in 
Kentucky. That this high stakes culture is both a creature and a master of Kentucky’s officials is 
probably an unanticipated consequence of systemic reform. The dual data controls and roles in 
Kentucky have had a strong effect on evaluation designs. 

The scope of Kentucky’s reform constrain design and projects. As with most sweeping 
generalizations, there is a kernel of truth to the KDE disclaimer that “the data in question could 
not, and maybe should not, be disaggregated.” When Kentucky’s legislature passed KERA in 
1990 , they were responding to the commonwealth’s educational problems with equity and 
adequacy. Finance, curriculum, and governance formed the three major components of reform. 
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Any one of these components alone could have made significant changes in Kentucky’s 
educational system, but the unifying piece is school level accountability through statewide 
assessment. Every programmatic strand of KERA is tied to supporting schools in meeting their 
accountability requirements (Legislative Research Commission, 1997; Task Force on Education 
Reform, 1989). Thus, any programmatic strand analysis is confounded by the systemic presence 
of other programmatic strands. It is difficult to isolate one component of the reform initiative 
and attribute student achievement changes specifically to that one. 

Synopsis of Findings 

Preliminary results suggest that the political agenda and organizational structure of the 
Kentucky Department of Educational influences evaluation processes. These results are not 
unexpected. The tension between researchers and practitioners has been documented elsewhere. 
Because of the impressive scope of Kentucky’s educational reform, evaluation studies are not 
only politically constrained, they are logistically constrained. The complexity of the reforms 
actually prevents comprehensive answers to the questions lawmakers most want answered. That 
is, researchers respond to the question: Is this initiative effective? with: How can we tell you if 
it’s effective when other initiatives confound the results? 

Conclusions 

The tension between evaluators and clients is genuine. Kentucky’s strains are typical. 
Researchers have made similar findings in smaller scale studies of how customer and social 
services are evaluated (Barabba & Zaltman, 1991; Brown-McGowan & Eichelberger, 1993). The 
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current study adds to that body of research on another dimension, that of statewide policy 
evaluation research. These results suggest that the political boundaries of statewide policy 
evaluation research are more complex and problematic than the generally intricate questions of 
reform effects. Moreover, these political borders may prevent adequate assessment of reform 
efforts to improve the conditions of teaching and learning. 
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